Repository navigation
Queries returning duplicate column names fail with ArrowInvalid: Can't unify schema with duplicate field names since 4.2.6 (#989) - #995
Conversation
peco-engineer-bot
Bot
added
the
engineer-bot
![peco-review-bot[bot]](https://avatars.githubusercontent.com/in/3801177?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict: 1 Low
Looks good — the fix correctly short-circuits to a non-promoting concat_tables when all chunk schemas are equal, which is exactly what's needed to avoid Arrow's duplicate-field-name unification error, and the empty-list/single-chunk cases are safe via the existing guard in concat_table_chunks. One low-severity note about a remaining edge (duplicate names + genuinely differing schemas still hit the promote path). Unit + e2e coverage added.
Addresses: - #4341406782 at src/databricks/sql/utils.py:940 Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good — the fix correctly short-circuits to unpromoted concat_tables when all chunk schemas are equal (handles the single-chunk and duplicate-name cases) while preserving the GH #418 promotion path when they differ, and the documented known-limitation is honest. One low note: the new e2e test lacks the @skipUnless(pysql_supports_arrow()) guard its sibling arrow tests use.
Addresses: - #4341457400 at tests/e2e/test_driver.py:456 Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
GitHub Actions
Failure
GitHub Actions
Failure
GitHub Actions
Failure
![peco-engineer-bot[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
…'t unify schema with duplicate field names` since 4.2.6 (#989) Signed-off-by: peco-engineer-bot[bot] <3815206+peco-engineer-bot[bot]@users.noreply.github.com>
Addresses: - #4341406782 at src/databricks/sql/utils.py:940 Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
Addresses: - #4341457400 at tests/e2e/test_driver.py:456 Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
The non-pyarrow unit test jobs import pyarrow as None, so the new test failed with AttributeError instead of being skipped like its neighbors. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Mehmet Can Kurt <mehmet.cankurt@databricks.com>
adfc573 to
e45ceac
Compare
Summary
Automated fix for #989 — Queries returning duplicate column names fail with
ArrowInvalid: Can't unify schema with duplicate field namessince 4.2.6.Fixed _concat_arrow_tables in src/databricks/sql/utils.py to concatenate chunks without schema promotion when all chunk schemas are equal, avoiding Arrow's schema unification which rejects duplicate field names (GH #989); promotion is retained only when schemas actually differ (the GH #418 nullability case). Verified green via a live-warehouse e2e test asserting duplicate-named columns survive fetchall_arrow()/fetchall()/fetchone(), plus a unit test on the concat path.
Root cause & plan
Root cause: src/databricks/sql/utils.py::_concat_arrow_tables unconditionally calls pyarrow.concat_tables(table_chunks, promote_options="default") (falling back to promote=True). Arrow's schema unification, triggered by any promote option, raises "ArrowInvalid: Can't unify schema with duplicate field names" — even for a single chunk where no promotion is needed. Every Arrow fetch on the Thrift backend routes through concat_table_chunks -> _concat_arrow_tables, so any query whose result has two columns with the same name (e.g. SELECT 1 AS a, 2 AS a, or SELECT * FROM a JOIN b with shared column names) fails on fetchall()/fetchone()/fetchall_arrow(). This regressed in 4.2.6 (PRs #751/#814 for #418); 4.2.5 returned Row(a=1, a=2) correctly. The connector otherwise supports duplicate names (Row docstring, _convert_arrow_table rename logic, unit test test_duplicate_column_names which only exercises _convert_arrow_table and never the concat path).
Files:
src/databricks/sql/utils.py,tests/e2e/test_driver.pyPlanned coverage:
Files changed
tests/e2e/test_driver.pysrc/databricks/sql/utils.pytests/unit/test_util.pyTest plan
tests/e2e/test_driver.py::TestPySQLCoreSuite::test_duplicate_column_names_survive_arrow_fetch— fails (red) against the original code, passes (green) after the fixtests/unit/test_util.py::TestUtils::test_concat_table_chunks_duplicate_column_names— fails (red) against the original code, passes (green) after the fix🤖 Generated by engineer-bot (bug-fix flow) — review before merge.